Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pybind] Add back extension suffix for portable lib shared object in pip wheel #6363

Closed
wants to merge 2 commits into from

Conversation

larryliu0820
Copy link
Contributor

@larryliu0820 larryliu0820 commented Oct 18, 2024

Summary: Initially our portable lib prebuilt library is named as:

_portable_lib.cpython-310-x86_64-linux-gnu.so

Where it includes an EXT_SUFFIX cpython-310-x86_64-linux-gnu consists of architecture and build OS information.

This is enforced by setuptools following PEP 3149 and there's no good way to change this behavior.

#5961 was an attempt to rename the .so filename at packaging stage to _portable_lib.so, so that it is easier to be found by find_package() macro in CMake.

However #5961 is breaking the other prebuilt libraries such as libcustom_ops_aot_lib.so which depends on the original _portable_lib.cpython-310-x86_64-linux-gnu.so name in its RPATH.

This PR is a fix that reverts #5961 and restore the full name during packaging, but try to match the EXT_SUFFIX in CMake to be able to find the .so file.

Test Plan:

EXECUTORCH_BUILD_PYBIND=ON python setup.py bdist_wheel # build wheel including _portable_lib.*.so
pip install dist/executorch-0.5.0a0+7510f8c-cp310-cp310-linux_x86_64.whl
python -c "from executorch.extension.llm.custom_ops import sdpa_with_kv_cache"

# Run torchao script to make sure find_package() works
push ao/torchao/experimental
bash build_torchao_ops.sh executorch

Does not throw _portable_lib.cpython-310-x86_64-linux-gnu.so not found error.

Reviewers:

Subscribers:

Tasks:

Tags:

…wheel

Summary: Initially our portable lib prebuilt library is named as:

```
_portable_lib.cpython-310-x86_64-linux-gnu.so
```
Where it includes an `EXT_SUFFIX` `cpython-310-x86_64-linux-gnu` consists of architecture and build OS
information.

This is enforced by `setuptools` following PEP 3149 and there's no good
way to change this behavior.

which is easier to be found by `find_package()` macro in CMake.

However #5961 is breaking the other prebuilt libraries such as
`libcustom_ops_aot_lib.so` which depends on the original
`_portable_lib.cpython-310-x86_64-linux-gnu.so` name in its RPATH.

This PR is a fix that reverts #5961 and restore the full name during
packaging, but try to match the `EXT_SUFFIX` in CMake to be able to find
the .so file.

Test Plan:

```bash
python -c "from executorch.extension.llm.custom_ops import
sdpa_with_kv_cache"
```

Does not throw `_portable_lib.cpython-310-x86_64-linux-gnu.so` not found
error.

Reviewers:

Subscribers:

Tasks:

Tags:
Copy link

pytorch-bot bot commented Oct 18, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6363

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures

As of commit 8885cdb with merge base 7510f8c (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 18, 2024
@larryliu0820 larryliu0820 changed the title [pybind] Add back extension suffix for portable lib shared object in … [pybind] Add back extension suffix for portable lib shared object in pip wheel Oct 18, 2024
@facebook-github-bot
Copy link
Contributor

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@facebook-github-bot
Copy link
Contributor

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@larryliu0820 merged this pull request in b3932c0.

@kirklandsign kirklandsign deleted the fix_nightly branch October 19, 2024 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants